Method and system for projecting a pattern in mixed reality

ABSTRACT

A method and a system for projecting a pattern onto a real surface, comprising: an image acquisition module; a video projector; a processor; and a memory comprising instructions that can be executed by the processor in order to: acquire an image on which a real surface is at least partially visible by means of said image acquisition module; determine a geometric transformation to be applied to said pattern such that once the pattern is projected by a video projector, it appears in a predefined location on the real surface, the determination of the geometric transformation taking into account the position of the video projector and the position of said real surface obtained by means of said acquired image; apply the geometric transformation to said pattern; and project the pattern by means of the video projector.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a National Stage Application of InternationalApplication No. PCT/FR2018/053259, filed on Dec. 13, 2018, now publishedas WO2019/115951 and which claims priority to French Application No.FR1762176, filed on Dec. 14, 2017.

BACKGROUND

The present disclosure relates to the general field of mixed reality,sometimes also referred to as hybrid reality, and relates moreparticularly to a method of projecting a pattern on a real surface.

Mixed reality is a technology that makes it possible to fuse a virtualworld with the real world, in order to produce a new environment andvisualizations in which physical objects of the real world, and digitalobjects of the virtual world, coexist and can possibly interact, forexample in real time. This technology is used more and more, inparticular in the field of entertainment, e.g. video games or in theworld of business, e.g. immersive learning content.

In order to obtain this fusion, it is possible to use any informationdisplay means, in particular a display screen, augmented reality ormixed reality glasses, such as the “Microsoft Hololens” glasses(registered trademark), or even a video projector.

An example of use of mixed reality is that of visualizing a virtual itemof furniture arranged such that it can be seen by a user at a locationin a room.

The current mixed reality techniques only allow for visualization ofobjects, and do not make it possible to implement a preview which wouldmake the performance of an action on a real object sufficiently easierfor a user.

There is therefore a need for new applications in mixed reality.

The disclosure aims in particular to overcome these disadvantages.

SUMMARY

The present disclosure meets this need by proposing a method ofprojecting a pattern on a real surface, the method comprising:

-   -   acquiring an image on which the real surface is visible at least        in part,    -   determining a geometric transformation to be applied to the        pattern such that, once projected by a video projector, the        pattern appears in a predefined position of the real surface,        i.e. in a predefined location and having a predefined shape or a        predefined operation, the determination of the geometric        transformation taking account of the position of the video        projector and of the position of the real surface obtained by        means of the acquired image,    -   applying the geometric transformation to the pattern,    -   projecting the pattern by means of the video projector. In other        words, projecting the pattern after the geometric transformation        has been applied, which makes it possible to ensure that the        pattern appears at the predetermined position of the real        surface.

In the present description, the term “position” refers both to thelocation and to the orientation. Alternatively, a person skilled in theart may generally use the term “pose.”

The surface is referred to as the real surface because it is not part ofa virtual world. Indeed, the real surface is a surface in the users'three-dimensional space, and thus has its own location in space. It canbe noted that the virtual world is a three-dimensional space whichexists only digitally.

The pattern exists initially in the virtual world, i.e. in digital form,for example a drawing recorded in the form of a raster or vector image.By being projected onto the real surface, the pattern appears in thereal world, and therefore the method as defined above is a mixed realitymethod.

The video projector may be in such a position that, when the pattern isprojected onto the real surface without geometric transformation, thepattern may appear in an undesired location or may even appear such thatit is deformed. This results in particular from the position of thevideo projector relative to that of the real surface, the opticalproperties of the video projector, etc.

The geometric transformation can therefore be determined such that thepattern appears in a predefined location. In examples, the patternappears having a shape (appearance) that is also predefined, such thatthe deformation or shift effects do not appear.

The location is predefined because it is selected, for example by auser, before the method is implemented.

A person skilled in the art knows how to implement image-processingtechniques on the acquired image, in order to determine how the realsurface is positioned.

Knowing the position of the video projector, and that of the realsurface, since these two positions are in the real world, it is possibleto obtain the geometric transformation.

Indeed, determining the geometric transformation can comprise a step ofregistering a virtual world relative to the real world, carried out onthe basis of detection of the real surface and possibly the position ofthe projector. This makes it possible to first of all determine, in thevirtual world, the way in which, from the perspective of the projectorin the virtual world, the pattern must appear on the real surface, inorder to deduce therefrom the transformation to be applied to thepattern, and to project the transformed pattern.

The method can make it possible to pre-visualize a drawing on the realsurface before this is carried out. In other words, the method is usedas a drawing aid.

For example, a pattern to be drawn is projected onto the real surface.For greater ease, the user turns the real surface. A new image may beacquired, and this makes it possible to determine that a rotation mustbe applied to the pattern in order to project the pattern such that itappears in the same location of the real surface. This facilitatescreation of a drawing on the real surface.

It will be noted that the real surface may be planar or non-planar.

According to one embodiment, an image is acquired on which at least aportion of a marking associated with the real surface is visible, the atleast one portion of the marking is detected, and the determination ofthe geometric transformation takes account of the detection of themarking.

For example, the marking may be printed, stuck or placed on the realsurface so as to be firmly associated therewith. A person skilled in theart knows to select a marking which can be detected on an acquiredimage, and which makes it possible to detect the position of the realsurface.

For example, the marking may be a square marking having a patterncomprising particular points. The marking may in particular comprise anon-symmetrical black-and-white pattern (or one having a high contrastratio).

By way of illustration, if the marking is in gray-scale, and the colorwhite is associated with a black intensity of 0% and the color black isassociated with a black intensity of 100%, then a marking may be usedthat has elements having a black intensity difference of at least 70points, which corresponds to a high contrast ratio. For example, it ispossible to use a gray at 15% black and a gray at 85% black, or a whiteat 0% black and a gray at 70% black.

It is also possible to use a colored pattern having colors associatedwith black intensities separated by 70 points. For example a pale yellowat 10% black and a dark blue at 80% black.

Alternatively, the marking may be a three-dimensional marking. Thisembodiment is particularly suitable for the use of an image acquisitionmodule of the RGB-D type, “Red, Green, Blue, and Depth”, which is anacronym that is well known to a person skilled in the art.

It can also be noted that the marking can be placed on a movable supportin order to be associated with the real object. For example, the supportmay be flexible and comprise means for attaching the support to itself,for example so as to surround the real surface or to the real surface.

It is possible in particular to select a marking which surrounds thereal surface if the surface is the surface of a real object, such thatthe marking is visible all around the object, so that the object can beturned.

According to one embodiment, the pattern is projected in a zone of thereal surface that is separate from that comprising the marking.

This particular embodiment facilitates the implementation of asubsequent step of drawing, because the marking and the elements thereofdo not hinder this step.

It can be noted that “separate” is intended, for example, to mean thatthe zone is spaced apart from the marking by at least a distance of theorder of a centimeter.

It can also be noted that using a marking makes it possible to moreeasily register a virtual world on the real world. Nonetheless, themethods or systems disclosed herein is in no way limited to the use of amarking, since some real surfaces (a person skilled in the art is ableto identify which) may have a detectable position due to the shapethereof (for example on the basis of the relative positions of thecorners of the surface).

The step of “registration” of a virtual world is intended for aligningthe virtual world and the real world. This in particular makes itpossible to facilitate the determination of a geometric transformationsuch that, once projected by a video projector, the pattern appears in apredefined location of the real surface.

The visible portion of the marking makes it possible to determine anorientation of the marking and also a dimension (for example a distancebetween two elements of the marking). From then on, the visible portionof the marking can be considered to be associated with a real worldreference point (for example an orthonormal reference point), and avirtual reference point will be developed on the basis of the real worldreference point. The virtual reference point thus developed makes itpossible to determine the geometric transformation.

According to a particular embodiment, at least a portion of anadditional marking associated with the real surface is furthermorevisible on the image, the at least one portion of the additional markingis detected, and the determination of the geometric transformation takesaccount of the detection of the marking.

Thus, the method is in no way limited to the use of a single marking,and it is possible to use one or more additional markings in order tofacilitate the detection of the real surface.

According to a particular embodiment, the geometric transformationcomprises a homography. The homography may comprise rotation and/ortranslation and/or homothety.

For example, a person skilled in the art knows to determine thehomography by means of determining a homothety coefficient, atranslation matrix and a rotation matrix, on the basis of knowledge ofthe position of the video projector, the detection of the position ofthe real surface, and finally the expected position of the projectedpattern.

According to a particular embodiment, the real surface is a surface of athree-dimensional real object, the method further comprising:

-   -   registering a virtual world on the basis of the image, possibly        by means of a marking,    -   placing a three-dimensional virtual object in the registered        virtual world,    -   applying the pattern to the three-dimensional virtual object,    -   the projection of the pattern comprising projection of the        pattern applied to the three-dimensional virtual object and that        as viewed from a perspective associated with the video        projector.

It can be noted that the perspective of the video projector can belocated approximately at the outlet of the optical system of the videoprojector.

It can be noted that the application of the pattern to thethree-dimensional virtual object can use techniques generallydesignated, by a person skilled in the art, as “texture mapping.”

A person skilled in the art in particular knows how to apply textures tovirtual three-dimensional objects such as three-dimensional meshes.

It will be noted that, since the three-dimensional virtual objectselected has a shape similar to that of the real three-dimensionalobject, the pattern will be projected as though it were applied to thereal surface. For example, the three-dimensional virtual object can beselected by a user, taking into account the shape of the real object.

This can in particular make it possible to facilitate the previewing ofa drawing corresponding to the pattern to be achieved on the realobject, and this can also make it possible to facilitate a step ofcreating a drawing that corresponds to the pattern on the real object.

Indeed, the method may comprise, following the display step, a displayof creating a drawing on the real object, it being possible for thedrawing to correspond to the pattern.

“As viewed from a perspective associated with the video projector” meansthat the projection was determined in the virtual world from thisperspective.

It will also be noted that, in this embodiment, determining thegeometric transformation comprises the steps of placing thethree-dimensional virtual object and applying the pattern. Thetranslation and rotation matrices can also be used in this case, as wellas a homothety coefficient. In other words, the geometric transformationis a homography.

According to a particular embodiment, the three-dimensional virtualobject is selected from a library of three-dimensional virtual objects.

Thus, a user can select a virtual object that has the shape closest tothat of the real three-dimensional object. The three-dimensional virtualobjects from the library of three-dimensional virtual objects may bemeshes that are well-known to a person skilled in the art, orthree-dimensional shapes associated with parameters. By way ofillustration, a cylindrical three-dimensional shape having a circularbase can be associated with a diameter parameter and a height parameter.

According to a particular embodiment, the method comprises a preliminarystep of adjusting the three-dimensional virtual object.

The preliminary adjustment step can be performed after priorimplementation of some steps of the method, in order to place a(non-adjusted) virtual object in the virtual world.

The preliminary adjustment step may comprise projecting thethree-dimensional virtual object (or of a mesh representing it) on thereal surface, or of displaying it on a screen in a manner superposed onthe acquired real world image. The projection of the three-dimensionalvirtual object can be achieved by projecting a partially opaquethree-dimensional virtual object such that it is possible to view boththe real object and the three-dimensional virtual object at the sametime. If the three-dimensional virtual object is not perfectly overlaidon the real object, the user can easily see the deviations between thethree-dimensional virtual object and the real object.

This adjustment may comprise a displacement, in the virtual world, ofthe three-dimensional virtual object, or a deformation of thethree-dimensional virtual object in the virtual world.

According to a particular embodiment, the adjustment step is implementedby means of receiving a command from a user.

By way of illustration, this step can be implemented by means of aninterface for the user.

According to a particular embodiment, the method comprises preliminarycalibration of an image acquisition module that is used for the imageacquisition, and in which an image is acquired on which a firstcalibration pattern is visible at least in part.

It is possible, for example, to use a first calibration pattern in theform of a checkerboard, in order to determine whether a distortionappears on the image acquired of the calibration pattern. This may makeit possible to determine whether a barrel distortion or pincushiondistortion appears when an image is acquired. These distortions areassociated with features of the image acquisition module used. A personskilled in the art knows to implement image processing in order that theacquired image is corrected and without distortions.

According to a particular embodiment, the method comprises preliminarycalibration of the video projector, and in which a second calibrationpattern is projected, and an image is acquired in which the secondcalibration pattern is visible at least in part.

It is possible, for example, to use a first calibration pattern in theform of a checkerboard (in the virtual world), in order to determinewhether a distortion appears on the image acquired of the projectedcalibration pattern. This may make it possible to determine whether abarrel distortion or pincushion distortion appears when an image isprojected. These distortions are associated with features of the videoprojector used. A person skilled in the art knows to implement imageprocessing in order that the projected image is corrected and withoutdistortions.

According to a particular embodiment, the method comprises a step ofdetecting the position of a writing instrument on the image.

In particular, it is possible to obtain the position of the point of thewriting instrument used.

This embodiment is of particular interest for applications in which auser reproduces the projected pattern using a writing instrument. Thismakes it possible, for example, to check that the writing instrument ispositioned in an expected position.

According to a particular embodiment, the detection of the position ofthe writing instrument is implemented by means of a detection patternthat is associated with the writing instrument and is visible on theimage, at least in part.

For example, it is possible to use a non-repetitive marking as thedetection pattern, the marking having a high contrast ratio, such thatwhen a portion of the detection pattern is visible on the acquiredimage, it is possible to determine the position of the writinginstrument.

According to a particular embodiment, an alert is generated if theposition of the writing instrument strays from an expected position.

This particular embodiment can comprise determination of a distancebetween the position of the writing instrument and, for example, theposition of the projected pattern.

This can make it possible to help the user draw the pattern, for exampleby generating projected visual alerts, sound alerts, or even alerts byvibration, in the writing instrument.

According to a particular embodiment, the method comprises a step ofdetecting a pattern drawn by the user on the real surface.

The drawn pattern may be visible, at least in part, on the acquiredimage, and it is therefore possible to detect the pattern, for examplein order to verify that it corresponds to the projected pattern.

It is also possible to detect the drawn pattern, in order to adapt theprojected pattern. For example, a complex pattern may be projected in aplurality of steps, the transition from one step to another taking placefollowing detection that the drawn pattern for a first part of thecomplex pattern has indeed been drawn.

According to a particular embodiment, an alert is generated if thepattern drawn by the user differs from an expected pattern.

This particular embodiment can comprise determination of distancesbetween the drawn pattern and the position of the projected pattern.

This can make it possible to help the user draw the pattern, for exampleby generating projected visual alerts, sound alerts, or even alerts byvibration, in the writing instrument.

It is also possible to attribute a score to the user, on the basis ofthe detection of the projected pattern, for the purpose of entertainmentuse.

The present disclosure also proposes a system for projecting a patternon a real surface, comprising:

-   -   an image acquisition module,    -   a video projector,    -   a processor,    -   a memory comprising instructions that can be executed by the        processor for:    -   acquiring, by means of the image acquisition module, an image on        which a real surface is visible at least in part,    -   determining a geometric transformation to be applied to the        pattern such that, once projected by a video projector, said        pattern appears in a predefined location of the real surface,        the determination of the geometric transformation taking account        of the position of the video projector and of the position of        the real surface obtained by means of the acquired image,    -   applying the geometric transformation to the pattern,    -   projecting the pattern by means of the video projector.

The system can be configured for implementing each embodiment of themethod as defined above.

According to a particular embodiment, the system further comprises amarking that is designed to be associated with the real surface, thememory further comprising instructions for detecting at least a portionof the marking, and for determining the geometric transformation, takingaccount of the detection of the marking.

The present disclosure also proposes a computer program comprisinginstructions for executing the steps of a method as defined above whenthe program is executed by a computer.

It should be noted that the computer programs mentioned in the presentdisclosure can use any programming language and be in the form of sourcecode, object code or intermediate code between source code and objectcode, such as in a partially compiled form or in any other desired form.

The disclosure also proposes a recording medium that can be read by acomputer on which a computer program is saved that comprisesinstructions for executing the steps of a test method as defined above.

The recording (or information) media mentioned in the present disclosuremay be any entity or device that is capable of storing the program. Forexample, the medium may comprise a storage means, such as a ROM, forexample a CD ROM or a ROM having a microelectronics circuit, or even amagnetic recording means, for example a floppy disc or a hard disc.

Alternatively, the recording media may correspond to a transmittablemedium such as an electronic or optical signal, which can be sent via anelectrical or optical cable, by radio, or by other means. The programmay in particular be downloaded from a network of the Internet type.

Alternatively, the recording media may correspond to an integratedcircuit in which the program is incorporated, the circuit being designedto carry out or to be used in the carrying out the method in question.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features and advantages of the present disclosure can be found inthe following description, with reference to the accompanying drawingswhich illustrate a non-limiting example.

In the figures:

FIG. 1 schematically shows the steps of a method for projecting apattern on a real surface,

FIG. 2 schematically shows a system for projecting a pattern, accordingto an embodiment,

FIGS. 3A and 3B show the projection of a pattern when the real surfaceis turned,

FIG. 4 shows a surface, viewed in an acquired image comprising markings,

FIGS. 5A and 5B show the projection of a pattern on a realthree-dimensional object,

FIG. 6 shows the detection of the position of a writing instrument,

FIG. 7 shows the detection of a drawn pattern, and

FIGS. 8A, 8B and 8C show calibration patterns.

DETAILED DESCRIPTION OF AN EMBODIMENT

A method and a system according to a particular embodiment will now bedescribed.

FIG. 1 schematically shows the steps of a method for projecting apattern on a real surface.

This method can be implemented by a system comprising a video projector,an image acquisition module such as a camera, and in addition a devicethat is capable of executing the computer program instructions,comprising a processor, for example a smartphone or a touchpad.

In this method, the pattern is a drawing which is intended to bedisplayed by means of projection on the real surface, as though it hadbeen applied to the real object. The real surface may be planar, or itmay be the surface of a three-dimensional object. This pattern isselected before the method is implemented.

In a first step E01, a real world image is acquired, on which the realsurface is visible at least in part.

“Visible at least in part” means that the real surface may be too largeto be in the acquired image. Alternatively, if the real surface is thesurface of a three-dimensional object, portions of the surface may behidden due to occlusion.

The real surface is in a position that can be determined by imageprocessing methods, for example if the surface is rectangular it ispossible to determine the position of the corners thereof.

Alternatively, the real surface can be associated with the imageacquisition module which acquires the image; the position of the realsurface is thus fixed, and it is not necessary to detect it on theacquired image.

As will be described in greater detail in the following, it is alsopossible to use one or more markings for detecting the position of thereal surface.

In a second step E02, a geometric transformation to be applied to thepattern is determined such that, once projected by a video projector,the pattern appears in a predefined location of the real surface. Thisdetermination takes into account the position of the video projector andthe position of the real surface, obtained by means of the image whichwas acquired during step E01.

By way of illustration, the geometric transformation can be a homographywhich can comprise a rotation, translation and, in examples, homothety.This step is necessary insofar as the video projector projects an imagethat can be deformed, according to the position of the video projectorrelative to the real surface.

In a step E03, the geometric transformation is applied to the pattern.

In step E04, the pattern to which the geometric transformation has beenapplied is projected.

Steps E01 to E04 can also be defined as computer program instructions.When the computer program is executed by a processor, it in particularcontrols an image acquisition module, and also a video projector.

FIG. 2 shows a system 200 for projecting a pattern on a real surface.

This system comprises an image acquisition module 201 of the cameratype. It can be noted that, alternatively, the image acquisition modulemay be an RGB-D image acquisition module. The image acquisition modulecomprises optical means (not shown here) which make it possible toobtain an image of a selected zone which will be described in thefollowing as being the real surface (or a portion thereof).

The system 200 also comprises a video projector 202 which is alsooriented such that it can project patterns on the same zone, referred toas the real surface.

In order to implement the steps of the method as described withreference to FIG. 1, the system 200 comprises a calculation unit 203which comprises a processor 204 and a memory 205. The memory 205 is anon-volatile memory, and the computer program instructions may have beenstored within the memory 205. In this case, computer programinstructions 206 a, 206 b, 206 c and 206 d have been stored in thememory 505 in order to implement, respectively, steps E01 to E04,described with reference to FIG. 1.

In particular, the instruction 206 a implements the step of acquiring animage of a real surface 207, shown in the figure while controlling theimage acquisition module 201.

The instruction 206 b implements the determination of a geometrictransformation of the pattern such that, once projected by a videoprojector, the projected pattern (denoted 208) appears in a predefinedlocation of the real surface 207, the determination of the geometrictransformation taking account of the position of the video projector andof the position of the real surface obtained by means of the acquiredimage.

The instruction 206 c applies the geometric transformation to thepattern.

The instruction 206 d controls the video projector so as to project thepattern after the geometric transformation has been applied.

The figure shows the projected pattern 207. The projection can help auser to reproduce the pattern on the surface.

For example, the user may use a writing instrument 209 to draw a pattern210, while attempting to follow the projected pattern.

It can be noted that, in the example of FIG. 2, the calculation unit203, the image acquisition module 201 and the projector 202 are allarranged in the same housing. From then on, the position of the videoprojector is always known.

Alternatively, the calculation unit 203, the image acquisition module201 and the projector 202 may be integrated in separate devices. Forexample, the calculation unit 203 may be integrated in a tablet or asmartphone, and the image acquisition module may be the camera of thetablet or of the smartphone.

FIG. 3A shows a system 300 for projecting a pattern. For this purpose, avideo projector 301 is used, which is designed to project patterns ontoa real surface 302.

In this example, the projected pattern 303 is a smiling face that isintended to be projected in the center of the real surface 302.

In order for the position of the real surface 302 to be known, the realsurface is provided with a marking 304. The marking can also be denoteda “target.” This term is used in particular in the context of thedevelopment kit (“SDK: Software Development Kit”) known as Vuforia(registered trade mark), marketed by the American company PTC. Indeed,the marking may be a “target” that is accepted by the Vuforiadevelopment kit for mixed reality, for example version 6, dating fromAugust 2016.

A marking of this kind makes it possible to perform a registration of avirtual world with respect to the real world, on the basis of thedetection of the marking.

The registration makes it easier to determine the geometrictransformation to be applied to the pattern in order for it to appear,as desired, in the center of the real surface 302.

In FIG. 3A, the elements 301 to 304 are shown from the perspective of animage acquisition module. Indeed, the acquired image 305 is also shownin the figure.

The initial pattern is in the form of a digital pattern 306, and hasalso been shown in the figure. The pattern can be selected by means of auser interface, for example an interface on a tablet.

Determining the transformation aims to define a rotation matrix, atranslation matrix, and, in examples, a homothety coefficient, whichmove the pattern from its original form (denoted 306) to the positionthat it has to assume in order that, projected by the video projector,it appears, as desired, in the center of the real surface 302. Byapplying the geometric transformation, the pattern to be projected 307is obtained.

FIG. 3B shows the system 300 after the real surface 302 has been turned.The position of the real surface will be determined again, on theacquired image 305, in order to derive therefrom a new geometrictransformation to be applied to the pattern 306. A pattern to beprojected 307 is again obtained.

Thus, a user wishing to be able to draw more easily can turn the realsurface 302, and the projected pattern 303 will also turn as soon as thesystem 300 has processed a new acquired image.

FIG. 4 shows a real surface 400 seen by an image acquisition module.

The real surface 400 comprises a plurality of markings 401 to 406 whichare all square in shape and have a black and white pattern.

The real surface, and the position of each of the markings, is knownbefore a method as described with reference to FIG. 1 is implemented. Inthis case, all the markings are different. As a result, detection of asingle marking makes it possible to know the position of the realsurface 400.

It can be noted that, in order to detect the markings, “thresholding”can be implemented, and then image processing can be implemented inorder to identify the square shapes which may be markings. It is thenpossible to analyze the squares in order to identify the markings.

FIG. 5A shows a system for projecting a pattern 500. The system isdesigned for projection, by means of a video projector 501, onto athree-dimensional object 502.

This figure is shown from the perspective of the image acquisitionmodule used to implement the method for projecting the pattern.

In this case, the three-dimensional object 502 is a parallelepiped, andtwo of the faces thereof comprise markings 504 and 505 so that they canbe identified on the acquired image.

In order to achieve good projection, a virtual parallelepiped can beselected from a library of three-dimensional objects. The object must bepositioned in the virtual world, instead and in place of the real objectin the real world.

Once it has been positioned, it is possible for the virtual object to beadjusted relative to the real object, for example by projecting thecontours (for example a mesh) of the virtual object in order to verifywhether they correspond to those of the real object.

Registration of the virtual world can be carried out on the basis of themarkings 504 and 505 visible on the acquired image.

In this case, it can be seen that the face of the real object comprisingthe marking 504 is in the field of the video projector 201, and it istherefore possible to project a portion of the pattern to berepresented, onto the face.

For this purpose, in the virtual world, the pattern (a flash oflightning) is applied to the three-dimensional virtual object.

Subsequently, the pattern applied to the object and that as viewed froma perspective associated with the video projector can be projected, andthe projected pattern 503 can be obtained. This amounts to determining ageometric transformation to be applied to the pattern. Indeed, it isconsidered in this case that the video projector is a camera in thevirtual world.

It can be noted that, in order to apply the pattern to thethree-dimensional virtual object, it is possible to use textureapplication techniques that are well known in the field ofthree-dimensional imagery. For example, it is possible to use theprogramming interface known as “OpenGL,” which is well known to a personskilled in the art, for example the version OpenGL 3.3.

FIG. 5B shows the system 500 after the user has turned the object 502.The face comprising the marking 505 is in the field of the videoprojector (although it was in shadow in FIG. 5A), and it is thereforepossible to project a second portion of the marking 503, in the form ofa smiling face. The face comprising the marking 504 is in shadow in theimage, and the marking 504 and the flash of lightning that has beenimplemented are shown transparent.

FIG. 6 shows an embodiment in which a writing instrument 600 is used todraw a pattern 601, while attempting to reproduce a projected pattern602.

The writing instrument 600 comprises a pattern 603 which, in this case,is in black and white, and which makes it possible to detect theposition of the writing instrument on an acquired image, during a methodof pattern projection and of implementation. Determining the position ofthe writing instrument allows for determination of the position of thepoint of the writing instrument.

From there, it is possible to determine the distance D1 between thepoint and the projected pattern 602. A person skilled in the art knowsto select an appropriate distance determination method.

If the distance D1 is greater than a predefined threshold, a warning canbe generated for the user. For example an audible or visual warning, orby vibration.

FIG. 7 shows a variant in which a writing instrument 700 has been usedto draw a pattern 701 in order to attempt to reproduce a projectedpattern 702.

The pattern which has already been drawn can be seen on the acquiredimage used for projecting the pattern 702 according to a method asdescribed above, and it is possible to determine the distance D2 betweenthe drawn pattern and the projected pattern. A person skilled in the artknows to select an appropriate distance determination method.

If the distance D2 is greater than a predefined threshold, a warning canbe generated for the user. For example an audible or visual warning, orby vibration.

In other applications, it is not necessary to determine the differencebetween the drawn pattern and the projected pattern. For example, theprojected pattern 702 can be projected only following detection that apattern 703 has been drawn.

This makes it possible to achieve complex drawings in a plurality ofsteps, or even to achieve drawings in a plurality of colors, bysimultaneously projecting the pattern portions associated with onecolor, before moving on to other pattern portions associated withanother color.

FIG. 8A shows a calibration pattern which can be used for calibratingthe image acquisition module and also for calibrating the videoprojector used for implementing a pattern projection method as definedabove.

The calibration pattern of FIG. 8A may for example have been printed ona real surface. It will then be observed on an image acquired by animage acquisition module.

The pattern may also be projected by the video projector. An image ofthe projected calibration pattern can also be acquired for calibrationpurposes.

FIG. 8B shows the barrel distortion which can be observed duringcalibration of the image acquisition module or during calibration of thevideo projector.

FIG. 8C shows the pincushion distortion which can be observed duringcalibration of the image acquisition module or during calibration of thevideo projector.

It is also possible to carry out image processing in order to correctthese two phenomena and to achieve a precise projection of the pattern.

The embodiments described above make it possible to achieve a preview ofa drawing on a real surface, in a manner that makes it possible to makeit easier to create a drawing on the real surface.

The invention claimed is:
 1. A method for projecting a pattern onto areal surface, the method comprising: acquiring an image on which thereal surface is visible at least in part, determining a geometrictransformation to be applied to the pattern such that, once projected bya video projector, the pattern appears in a predefined position of thereal surface, wherein determining of the geometric transformation takesinto account a position of the video projector and the position of thereal surface obtained by means of the acquired image, applying thegeometric transformation to the pattern, projecting the pattern usingthe video projector; and detecting the position of a writing instrumenton the acquired image, wherein the detection of the position of thewriting instrument is implemented by detecting a detection pattern thatis on an outer surface of the writing instrument and is visible on theacquired image, at least in part, wherein the method further comprisesdetecting a pattern drawn by a user on the real surface, wherein analert is automatically generated in real time if the position of thepattern drawn by the user differs from a position of an expectedpattern.
 2. The method according to claim 1, wherein the acquired imageincludes at least a portion of a marking associated with the realsurface that is visible, the at least one portion of the marking beingdetected, and determining of the geometric transformation taking accountof the detection of the marking.
 3. The method according to claim 2,wherein the pattern is projected in a zone of the real surface that isseparate from that comprising the marking.
 4. The method according toclaim 2, wherein at least a portion of an additional marking associatedwith the real surface is also visible on the acquired image, the atleast one portion of the additional marking being detected, anddetermining of the geometric transformation taking account of thedetection of the additional marking.
 5. The method according to claim 1,wherein the geometric transformation comprises a homography.
 6. Themethod according to claim 1, wherein the real surface is a surface of athree-dimensional real object, the method further comprising:registering a virtual world on the basis of the acquired image, placinga three-dimensional virtual object in the registered virtual world,applying the pattern to the three-dimensional virtual object, whereinthe pattern is a first pattern, and projection of the first patterncomprises projection of the first pattern applied to thethree-dimensional virtual object and that as viewed from a perspectiveassociated with the video projector when a first marking on the firstsurface of the three-dimensional real object is detected in the acquiredimage, and wherein the first pattern is projected on to the firstsurface of the three-dimensional real object, and the method includesapplying a second pattern to the three-dimensional virtual object,wherein the second pattern is different than the first pattern, themethod includes projecting the second pattern onto a second surface ofthe three-dimensional real object that is offset from the first surface,wherein the second pattern is projected when a second marking on thesecond surface is detected in the acquired image, wherein the secondmarking is different than the first marking.
 7. The method according toclaim 6, wherein the three-dimensional virtual object is selected from alibrary of three-dimensional virtual objects.
 8. The method according toclaim 6, further comprising a preliminary step of adjusting thethree-dimensional virtual object.
 9. The method according to claim 1,further comprising preliminary calibration of an image acquisitionmodule that is used for the image acquisition, and wherein an image isacquired on which a first calibration pattern is visible at least inpart.
 10. The method according to claim 1, further comprisingpreliminary calibration of the video projector, and wherein a secondcalibration pattern is projected, and an image is acquired in which thesecond calibration pattern is visible at least in part.
 11. A method forprojecting a pattern onto a real surface, the method comprising:acquiring an image on which the real surface is visible at least inpart, determining a geometric transformation to be applied to thepattern such that, once projected by a video projector, the patternappears in a predefined position of the real surface, whereindetermining of the geometric transformation takes into account aposition of the video projector and the position of the real surfaceobtained by means of the acquired image, applying the geometrictransformation to the pattern, projecting the pattern using the videoprojector, and detecting a pattern drawn by a user on the real surface,wherein an alert is automatically generated in real time if the positionof the pattern drawn differs from a position of an expected pattern. 12.A system for projecting a pattern on a real surface, the systemcomprising: an image acquisition module, a video projector, a processor,a memory comprising instructions that can be executed by the processorfor: acquiring, by the image acquisition module, an image on which areal surface is visible at least in part, determining a geometrictransformation to be applied to the pattern such that, once projected bya video projector, the pattern appears in a predefined location of thereal surface, wherein determining of the geometric transformation takesaccount of the position of the video projector and of the position ofthe real surface obtained from the acquired image, applying thegeometric transformation to the pattern, and projecting the patternusing the video projector, wherein the pattern is a first pattern, onlyafter detecting that the first pattern has been drawn onto the realsurface, projecting a second pattern, different from the first pattern,using the video projector.
 13. The system according to claim 12, furthercomprising a writing instrument having a non-symmetrical and highcontrast marking, the memory further comprising instructions fordetermining a location of the writing instrument taking account of thedetection of the marking.
 14. A non-transitory computer-readable storagemedium for storing instructions that, when executed by one or moreprocessors, perform the method according to claim
 1. 15. Anon-transitory computer-readable storage medium for storing instructionsthat, when executed by one or more processors, perform the methodaccording to claim
 11. 16. The method of claim 1, wherein the alert isautomatically generated in real time when a distance between the patterndrawn by the user and the expected pattern exceeds an expectedthreshold.
 17. The method of claim 1, wherein the detection patternincludes a high contrast pattern on an outer surface of the writinginstrument.
 18. The method of claim 1, wherein the method furthercomprises detecting a pattern drawn on the real surface, wherein analert is automatically generated in real time if the position of thepattern drawn by the user differs from an expected pattern, wherein thealert includes causing a vibration in the writing instrument orgenerating an audio alert, or generating a visual alert onto the realsurface, wherein the alert is automatically generated in real time whena distance between the pattern drawn by the user and the expectedpattern exceeds an expected threshold, wherein the detection patternincludes a high contrast pattern on an outer surface of the writinginstrument.
 19. The method of claim 1, wherein the pattern is a firstpattern, and the method includes: detecting whether the first patternhas been drawn onto the real surface, only after detecting that thefirst pattern or an approximation of the first pattern has been drawnonto the real surface, projecting a second pattern onto the realsurface, detecting whether the second pattern has been drawn onto thereal surface, only after detecting that the second pattern or anapproximation of the second pattern has been drawn onto the realsurface, projecting a third pattern onto the real surface, wherein thefirst pattern, the second pattern, and the third pattern are differentfrom one another.